In underwater acoustic communication, multipath effect channel can cause severe Inter-Symbol Interference (ISI). In view of the problems of the existing equalization algorithms when dealing with ISI, including slow convergence speed and huge steady-state error, as well as the complicated algorithm and being difficult to carry out hardware migration, a new variable step Least Mean Square (LMS) algorithm was proposed with anticosine step function and three adjustment parameters within the Feed-Forward Equalizer and Decision Feed-back Equalizer (FFE-DFE) structure. Firstly, simulations of three adjustment parameters including α, β, r were given to optimize the algorithm and compare it with traditional LMS algorithm, Modified Arctangent based Variable Step LMS (MA-VSLMS) and Hyperbolic Secant function based Variable Step size LMS algorithm (HS-VSLMS) in convergence and steady-state error. The simulation results showed that compared with the traditional LMS algorithm, the convergence speed of the proposed algorithm was 57.9% higher, and the steady-state error was reduced by 2 dB; compared with HS-VSLMS and MA-VSLMS, the convergence speed of the proposed algorithm was 26.3% and 15.8% higher, respectively, and the steady-state error was reduced by 1-2 dB. Finally, the proposed algorithm was transplanted to signal processing module and tested in an underwater experiment. Experimental results indicate that the signal is recovered very well after the equalizer, and the ISI problem caused by multipath effect is solved in the actual scene.